We study prime algebras of quadratic growth. Our first result is that if $A$ is a prime monomial algebra of quadratic growth then $A$ has finitely many prime ideals $P$ such that $A/P$ has GK dimension one. This shows that prime monomial algebras of quadratic growth have bounded matrix images. We next show that a prime graded algebra of quadratic growth has the property that the intersection of the nonzero prime ideals $P$ such that $A/P$ has GK dimension 2 is non-empty, provided there is at least one such ideal. From this we conclude that a prime monomial algebra of quadratic growth is either primitive or has nonzero locally nilpotent Jacobson radical. Finally, we show that there exists a prime monomial algebra $A$ of GK dimension two with unbounded matrix images and thus the quadratic growth hypothesis is necessary to conclude that there are only finitely many prime ideals such that $A/P$ has GK dimension 1.